Intel® Dynamic Load Balancer Driver for VMware ESXi* with DirectPath I/O
========================================================================

The Intel® Dynamic Load Balancer (Intel® DLB) is a PCIe device that provides
load balanced, prioritized scheduling of events across CPU cores enabling
efficient core-to-core communication. It is a hardware accelerator located
inside the latest Intel® Xeon® devices offered by Intel.

This software enables Single Root I/O Virtualization (SR-IOV) for the Intel® DLB
on VMware ESXi* hypervisor. SR-IOV enables the creation of Virtual Functions (VFs)
from a single Intel DLB Physical Function (PF) to support hardware acceleration
for multiple guest Virtual Machines (VMs).

This document describes how to install Intel® DLB SR-IOV Driver on VMware ESXi*
hypervisor and how to make Intel® DLB HW available inside VMs via DLB VFs.
For more details on how to use Intel® DLB from a VM, please refer to the
documents mentioned in the "Documentation" section.

Note that this document assumes that the reader is familiar with virtualization
technologies and has some level of familiarity with the VMware ESXi* hypervisor.
This document does not explain how to install VMware ESXi*, how to create a VM
and how to administrate hypervisor using vSphere* Host Client or VMware vCenter*.
For more details, please refer to VMware documentation.

*Other names and brands may be claimed as the property of others.


License
=======
Refer to LICENSE.txt in this package for license information before using this SW.


Details/Limitations of this Release
===================================
 * This software release is intended for platforms that contain
      4th Generation Intel® Xeon® Scalable Processor (2710 DLB)
      5th Generation Intel® Xeon® Scalable Processor (2710 DLB)
 * This driver release doesn't support VMware vSphere* vMotion
 * Number of PCI passthrough devices per VM is limited
      https://kb.vmware.com/s/article/1003497


Documentation
=============
Associated Linux SW and collaterals can be found on the Intel® Download Center
    https://www.intel.com/content/www/us/en/download/686372


Install Intel® DLB VMware SR-IOV Driver
=======================================
1. Open a SSH connection to the target ESXi host (SSH needs to be enabled).

2. Copy the driver package to the ESXi
   (assume here and further that the target location is in '/tmp' folder and
   "scp" utility is used to copy the file to the target ESXi at 10.10.10.10)
    > scp dlb_*_rel_bin_*.tar.gz root@10.10.10.10:/tmp

3. Extract the driver package
    > cd /tmp
    > tar -xzf dlb_*_rel_bin_*.tar.gz

4. Install the driver from component
    > esxcli software component apply -d /tmp/*Intel-dlb*.zip

5. Reboot the system to complete driver installation
    > reboot

6. Configure module parameters if needed. Please refer to these sections
    * Configure module parameters
    * DLB resource provision per VF

7. Verify the driver is loaded after reboot
    > esxcfg-module -l dlb | grep dlb
      dlb                      8    184
    > lspci -p | grep 2710
      0000:eb:00.0 8086:2710 8086:0000                V dlb
      0000:f0:00.0 8086:2710 8086:0000                V dlb
      0000:f5:00.0 8086:2710 8086:0000                V dlb
      0000:fa:00.0 8086:2710 8086:0000                V dlb


Configure module parameters
===========================
1. To list module parameters along with their descriptions
    > esxcli system module parameters list -m dlb

2. To restore module parameters to default values
    > esxcli system module parameters clear -m dlb

3. To set one module parameters
    > esxcli system module parameters set -m dlb -p name=value

4. To append a module parameter
    > esxcli system module parameters set -m dlb -a -p name=value

5. Driver needs to be reloaded to apply changes
    > esxcfg-module -u dlb
    > kill -HUP $(cat /var/run/vmware/vmkdevmgr.pid)

Note: If driver is used together with Intel® Accelerator Management Daemon for
VMware ESXi*, driver unload may report that "module symbols in use".  In this
case please proceed with system reboot or refer to daemon documentation on how
to temporally stop it.


DLB resource provision per VF
=============================
Amount of resources provisioned to VF is configured via module parameters
    > esxcli system module parameters set -m dlb -a -p provision_scale_factor=value

The "value" valid range is [0 - 16]. Default: 1. Recommended: 1, 2, 4.
0: No DLB provision per VF by default. Or custom specified by DLB tool/plugin.
M: DLB provision per VF is 1/M of all DLB resources, M is from [1, 16] range.


Enable SR-IOV
=============
1. Login to the vSphere Host Client on the target ESXi host.

2. In the left pane, expand "Host" and select "Manage".

3. In the center pane, on the "Hardware" tab, select "PCI devices".

4. Find devices with "Intel Corporation DLB PF" text in "Description" and click on
   "Configure SR-IOV" button.

5. In dialog window change "Enabled" param to "Yes" and enter the desired
   number of VFs in the range between 1 and maximum indicated in the window.
   Click "Save" to reconfigure DLB PF device with requested number of VFs.

6. Verify DLB VFs have been enabled finding them in "Description" by text
   "Intel Corporation DLB VF" (appear after reboot only) or by BDFs. The listed BDFs
   need to match with the output of the "lspci". Example (BDFs are 6d:00.x):
     > lspci -vn | grep 2711
       0000:6d:00.1 Class 0b40: 8086:2711 [PF_0.109.0_VF_0]
       0000:6d:00.2 Class 0b40: 8086:2711 [PF_0.109.0_VF_1]
       0000:6d:00.3 Class 0b40: 8086:2711 [PF_0.109.0_VF_2]
       0000:6d:00.4 Class 0b40: 8086:2711 [PF_0.109.0_VF_3]
       0000:6d:00.5 Class 0b40: 8086:2711 [PF_0.109.0_VF_4]

Notes:
 * VMware ESXi keeps SR-IOV information even if driver is removed.
 * The recommended number of VFs for VF resource provision scale factor 1 is 5.


Pass-through the PCI Device
==========================
1. Login to the vSphere Host Client on the target ESXi host.

2. In the left pane, expand "Virtual Machines" and select desired VM. Ensure
   that the VM is powered off.

3. Click on the "Edit" button to edit VM settings. A pop-up window will appear.

4. All memory in the VM must be reserved. Expand "Memory" and set
   "Reserve all guest memory" checkbox.

5. Click on "Add other device" and select "PCI device". The new PCI device will
   be added. The first VF in the system is selected by default but it could be
   not a DLB one. To select DLB VF, click the drop-down list and choose device
   with "Intel Corporation DLB VF" in name (appear after reboot only) or by BDFs.
   The listed BDFs need to match with the output of the "lspci -vn | grep 2711".
   See example above. Additional VFs can be added by repeating this step.

6. Click "Save".


Uninstall Intel® DLB Driver
===========================
1. Open a SSH connection to the target ESXi host.

2. Reset module parameters to default values (not required but recommended)
    > esxcli system module parameters clear -m dlb

3. Unload module (not required and may fail if daemon is used)
    > esxcfg-module -u dlb

4. Uninstall driver
    > esxcli software component remove -n Intel-dlb

5. Reboot the system to complete driver uninstallation
    > reboot

Note: It's not mandatory to reset module parameters but ESXi keeps this
      information after driver is removed from the system.
